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Abstract: This report presents the usage results of Grid'5000 over year 2008. Usage of the 
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Une annee dans la vie d'un systeme distribue 
experimental a grande echelle : la plate-forme Grid'5000 

en 2008 

Resume : Ce rapport preseiite les resultats d'usage de la plateforme experimentale Grid'5000 
pendant I'annee 2008. L'usage des principaux sites operationnels de Grid'5000 (Bordeaux, 
Lille, Lyon, Nancy, Orsay, Rennes, Sophia- Antipolis. Toulouse) est presente et analyse. 

Mots-cles : Systeme distribue a grande echelle, usage, Grid'5000 
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1 Introduction 

Some previous work on operational Grids [IDE+06] show that grids are not utihzed at 
their full capacity. We focus on the utilization of a large-scale experimental distributed 
system by relying on the case study of Grid'5000[Cea05]^, a French experimental Grid. We 
collected and analyzed the logs of the Grid'5000 platform for a full year of usage (2008). 
We present the main statistics observed for each site and some specific and representative 
computing nodes. A special analysis is dedicated to the grid usage of this experimental 
platform. This usage analysis has been used in some works on energy-efficient experimental 
distributed infrastructures [OLGOSa, OLGOSb]. A similar analysis has been done for 2007 
logs of Grid'5000 and can been found in [OL09]. 

This paper briefly presents the Grid'5000 experimental platform and the followed method- 
ology in Section 2. The usage statistics are then presented for each Grid'5000 site (Section 3). 

2 Definitions and Methodology 

2.1 Grid'5000: a Large-Scale Experimental Distributed System 

The Grid'5000 platform is an experimental testbed dedicated for research in computer sci- 
ence, made up of more than 3400 processors geographically distributed on 9 sites in France 
(see Fig. 1). This platform can be defined as a highly reconfigurable, controllable, and 
monitorable experimental large-scale distributed system. 




Figure 1: The Grid'5000 map 



^Some experiments of this article were performed on the Grid'5000 platform, an initiative from the 
French Ministry of Research through the ACI GRID incentive action, INRIA, CNRS and RENATER and 
other contributing partners (http://www.grid5000.fr). This research is supported by the GREEN-NET 
INRIA Cooperative Research Action: http://www.ens-lyon.fr/LIP/RESO/Projects/GREEN-NET/ 
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The utilization of Grid'5000 can be very specific. Eacli user can reserve (in advance) 
some computing nodes. During its reservation time, tlie user can be root on his reserved 
nodes and he can deploy his own system images, collect data, launch applications, reboot 
the nodes, and so on. The nodes are entirely dedicated to the user during his reservation. 

2.2 Experiment Methodology 

The user gives the resource manager (at least) a start time, a duration, and the number 
of required resources. These characteristics define the job (a reservation). The resource 
manager is in charge of the job's acceptance. It verifies if that job is compatible with 
previously accepted jobs and, if it is, it gives a job id to the user. 
There are three different type of jobs: 

• deploy: the user deploys its own environment, 

• default: the user uses the default environment. 

• hesteffort: they get scheduled on processes when no other job use them and these jobs 
are killed when a regular (deploy or default) job recently submitted needs the nodes 
used by a besteffort job. 

When a resource is not available for the user, it can be in different states: 

• dead: the resource is down (due to a component failure for example); 

• suspected: the resource does not work properly; 

• absent: the resource is not available for the user (not physically present). 

Moreover, the platform has changed between the beginning and the end of the measure- 
ments. For that reason, we present the results as percentages of the platform's capacity 
at the time of measurement (100% at a given time does not represent the same number of 
resources as 100% at another time). 

In order to obtain the utilization traces, we have used a history function provided by 
OAR^ [CCG+OS] called oarstat. This function provides the user with all the events that 
occur during a time period. An event can be a job or a problem on a resource (it is dead, 
absent or suspected) for a given period. Our goal is to obtain a detailed overview of the 
usage of an experimental grid. 

3 Results per Grid'5000 site 

For each Grid'5000 site, we provide several values and figures which represent the global 
usage of the site. All the times are given in seconds. A resource is a core and a job is 
actually a reservation. 

We have split the proposed statistics in three categories: 

^OAR is a resource manager (or batch scheduler) for large clusters. (http : //oar . imag. fr/index .html) 
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• the "platform and resources" part which hsts the available number of resources and 
some statistics on the resources' states. We also compute the real work time of the 
Grid'5000 site. The "real" percentage of work time is calculated without taking into 
account dead time or absent time (i.e. work time over total time minus dead time and 
absent time for all the resources). 

• the "jobs" part proposes some statistics on the number of submitted jobs (reservations), 
the mean time and mean number of resources per job. and the maximal duration of a 
job on a site. We also list the number of reservations used for deploying system images 
(deploy jobs). 

• the "users" section contains the number of users and the impact of users from other 
Grid'5000 sites. The percentage of users coming from other sites does not take into 
account any consideration of proportionality in terms of jobs. It is just the number 
of users coming from other sites over the total number of users (each user is counted 
once) . 

For each site, we provide four diagrams. The first one represents the weekly repartition 
in time of the resources between the different states: in red when some cores are dead {Dead 
state), in orange when they are suspected {Suspected state), in yellow when they are absent 
{Absent state), in green when they are working (a job is running) ( Work state), and in white 
when they are unoccupied (no job, no other state) {Idle state). 

The two other diagrams present the weekly time repartition of two particular resources: 
the median and the maximal resources. The "maximal resource" is the resource which has 
the maximal work time among the resources which are present for the whole 2008 year. The 
"median resource" is a resource which is present for the whole 2008 year and is the nearest 
to the median value of cumulative work over the experiment's duration. 

3.1 Usage of Grid'5000's Bordeaux site in 2008 

• Platform and resources: 

— Maximal number of resources (cores): 650 

— Mean time spent in each state for all the resources, in percentage: 

* Dead: 3.48% 

* Suspected: 5.53% 

* Absent: 4.06% 

* Work: 49.19% 

— Real percentage of work time (without taking into account the time when the 
resources are dead or absent): 53.20% 

• Jobs: 

— Number of jobs (reservations): 356222 
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— Mean time of a job: 2473.38 (41 minutes and 13 seconds) 

— Maximal dm'ation: 794428 s. (9 days 4 hours 40 minutes and 28 seconds) for job 
number 407783 

— Mean number of resources (cores) per job: 7.44 

— Percentage of deploy jobs: 0.97% 

— Percentage of time spent in deploy jobs compared to the work time: 22.32% 

— Percentage of jobs coming from other sites: 83.28% 

— Percentage of besteffort jobs: 58.03% 

— Percentage of time spent in besteffort jobs compared to the work time: 34.90% 
• Users: 

— Number of users: 223 

— Percentage of users coming from other sites: 87% 




RR n° 7481 



8 



Orgerie & Lefevre 



Distribution of unoccupied time and activity per weel< lor Bordeaux 
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Figure 3: Global diagram without dead time for Grid'5000's Bordeaux site 



Distribution in time of the median resource's states per week for Bordeaux - number 324 
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Figure 4: Median resource diagram for Grid'5000's Bordeaux site 
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Distribution in time of thie maximal resource's states per week for Bordeaux - number 526 
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Figure 5: Maximal resource diagram for Grid'5000's Bordeaux site 



JL We can see on these graphs that the site load is not related to the number of jobs. 
Indeed, between May and August, there are few jobs but the percentage of activity is high 
(Figure 2). Bordeaux is the less used Grid'5000 site for 2008 in terms of real activity. 
However, its activity has increased compared to the previous year [OL09]. 



3.2 Usage of Grid'5000's Lille site in 2008 

• Platform and resources: 

— Maximal number of resources (cores): 618 

— Mean time spent in each state for all the resources, in percentage: 

* Dead: 16.21% 

* Suspected: 4.67% 

* Absent: 8.61% 

* Work: 54.80% 

^ Real percentage of work time (without taking into account the time when the 
resources are dead or absent): 72.89% 

• Jobs: 

— Number of jobs (reservations): 344538 

— Mean time of a job: 3154.58 s. (52 minutes and 34 seconds) 

— Maximal duration: 756077 s. (8 days 18 hours 1 minute and 17 seconds) for job 
number 693546 

— Mean number of resources (cores) per job: 8.11 
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— Percentage of deploy jobs: 0.61% 

— Percentage of time spent in deploy jobs compared to the work time: 7.24% 

— Percentage of jobs coming from other sites: 38.97% 

— Percentage of besteffort jobs: 19.01% 

— Percentage of time spent in besteffort jobs compared to the work time: 37 
• Users: 

— Number of users: 198 

— Percentage of users coming from other sites: 65.66% 



Distribution in time of ttie different resource's stales per weel( for Liiie 
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Figure 6: Global diagram with dead time for Grid'5000's Lille site 
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Distribution of unoccupied time and activity per weei< lor Lilie 
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Figure 7: Global diagram without dead time for Grid'5000's Lille site 




Figure 8: Median resource diagram for Grid'5000's Lille site 
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Maximal resource's states per week for Lille - number 175 
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Figure 9: Maximal resource diagram for Grid'5000's Lille site 

The number of resources has greatly increase from 250 to 618 cores during the year 
2008. The last two months of the year show a really high utilization of this site (over 95% 
during those two months). Moreover, we can see on Figure 8 that even the oldest resources 
are reserved and used for experiments on Grid'5000 Lille site. 

3.3 Usage of Grid'5000's Lyon site in 2008 

• Platform and resources: 

— Maximal number of resources (cores): 322 

— Mean time spent in each state for all the resources, in percentage: 

* Dead: 1.82% 

* Suspected: 2.78% 

* Absent: 3.43% 

* Work: 65.64% 

^ Real percentage of work time (without taking into account the time when the 
resources are dead or absent): 69.27% 

• Jobs: 

— Number of jobs (reservations): 138217 

— Mean time of a job: 3723.55 s. (1 hour 2 minutes and 4 seconds) 

— Maximal duration: 1080001 s. (12 days 12 hours and 1 second) for job number 
124333 

— Mean number of resources (cores) per job: 4.39 
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— Percentage of deploy jobs: 3.88% 

— Percentage of time spent in deploy jobs compared to the work time: 49.69% 

— Percentage of jobs coming from other sites: 87.07% 

— Percentage of besteffort jobs: 44.00% 

— Percentage of time spent in besteffort jobs compared to the work time: 24.14% 
• Users: 

— Number of users: 169 

— Percentage of users coming from other sites: 73.37% 



Distribution in time of ttie different resource's states per weel< for Lyon 




Figure 10: Global diagram with dead time for Grid'5000's Lyon site 
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Figure 11: Global diagram without dead time for Grid'5000's Lyon site 



Median resource's states per week for Lyon - number 28 
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Figure 12: Median resource diagram for Grid'5000's Lyon site 
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Maximal resource's states per week for Lyon - number 195 




Figure 13: Maximal resource diagram for Grid'5000's Lyon site 



X Lyon is the smallest Grid'5000 site for the year 2008. No machine has been added 
during this year to this site. However, it is really used. Another peculiarity of this site is 
that it gets the smallest reservations in terms of mean number of resources per reservation 
and among the smallest reservations in terms of mean length. It also gets the biggest time 
spent (in percentage) in deploy jobs: almost half of the time nodes are working with user's 
environments. 



3.4 Usage of Grid'5000's Nancy site in 2008 

• Platform and resources: 

— Maximal number of resources (cores): 574 

— Mean time spent in each state for all the resources, in percentage: 

* Dead: 0.11% 

* Suspected: 0.90% 

* Absent: 1.95% 

* Work: 58.84% 

— Real percentage of work time (without taking into account the time when the 
resources are dead or absent): 60.08% 

• Jobs: 

— Number of jobs (reservations): 74592 

— Mean time of a job: 8912.82 s. (2 hours 28 minutes and 33 seconds) 

— Maximal duration: 1695503 s. (19 days 14 hours 58 minutes and 23 seconds) for 
job number 213465 
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— Mean number of resources (cores) per job: 14.63 

— Percentage of deploy jobs: 4.23% 

— Percentage of time spent in deploy jobs compared to the work time: 29.95% 

— Percentage of jobs coming from other sites: 20.11% 

— Percentage of besteffort jobs: 76.16% 

— Percentage of time spent in besteffort jobs compared to the work time: 17.05% 
• Users: 

— Number of users: 157 

— Percentage of users coming from other sites: 83.44% 



Distribution in time of tlie different resource's states per weel< for Nancy 
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Figure 14: Global diagram with dead time for Grid'5000's Nancy site 
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Distribution of unoccupied time and activity per weei< lor Nancy 
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Figure 15: Global diagram without dead time for Grid'5000's Nancy site 



Median resource's states per week for Nancy - number 568 
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Figure 16: Median resource diagram for Grid'5000's Nancy site 
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Maximal resource's states per week for Nancy - number 10 
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Figure 17: Maximal resource diagram for Grid'5000's Nancy site 

The amount of available resources remains constant during the year 2008. This site 
presents a healthy working: almost no dead, suspected and absent periods. However, the 
load is not really balanced between the nodes as shown by Figures 16 and 17. 

3.5 Usage of Grid'5000's Orsay site in 2008 

• Platform and resources: 

— Maximal number of resources (cores): 684 

— Mean time spent in each state for all the resources, in percentage: 

* Dead: 5.82% 

* Suspected: 12.20% 

* Absent: 6.45% 

* Work: 50.73% 

— Real percentage of work time (without taking into account the time when the 
resources are dead or absent): 57.82% 

• Jobs: 

— Number of jobs (reservations): 92862 

— Mean time of a job: 6246.07 s. (1 hour 44 minutes and 6 seconds) 

— Maximal duration: 1060487 s. (12 days 6 hours 34 minutes and 47 seconds) for 
job number 101189 

— Mean number of resources (cores) per job: 14.58 
^ Percentage of deploy jobs: 3.59% 
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— Percentage of time spent in deploy jobs compared to the work time: 38.98% 

— Percentage of jobs coming from other sites: 96.28% 

— Percentage of bestefFort jobs: 38.91% 

— Percentage of time spent in besteffort jobs compared to the work time: 27.17% 
• Users: 

— Number of users: 172 

— Percentage of users coming from other sites: 76.16% 





Figure 18: Global diagram with dead time for Grid'SOOO's Orsay site 



RR n° 7481 



20 



Orgerie & Lefevre 



Distribution of unoccupied time and activity per weei< lor Orsay 
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Figure 19: Global diagram without dead time for Grid'5000's Orsay site 



Median resource's states per weel< for Orsay - number 433 
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Figure 20: Median resource diagram for Grid'5000's Orsay site 
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Maximal resource's states per week for Orsay - number 421 
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Figure 21: Maximal resource diagram for Grid'5000's Orsay site 



X This site is really more used than the previous year (18.88% of real activity in 
2007 [OL09]). It is one of the biggest sites in terms of resources and attracts a lot of jobs 
coming from other sites. 



3.6 Usage of Grid'5000's Rennes site in 2008 

• Platform and resources: 

— Maximal number of resources (cores): 714 

— Mean time spent in each state for all the resources, in percentage: 

* Dead: 4.79% 

* Suspected: 0.44% 

* Absent: 0.79% 

* Work: 60.98% 

^ Real percentage of work time (without taking into account the time when the 
resources are dead or absent): 64.58% 



• Jobs: 



— Number of jobs (reservations): 58843 

^ Mean time of a job: 7069.33 s. (1 hour 57 minutes and 49 seconds) 

— Maximal duration: 604825 s. (7 days and 25 seconds) for job number 289263 

— Mean number of resources (cores) per job: 27.32 

— Percentage of deploy jobs: 7.84% 

— Percentage of time spent in deploy jobs compared to the work time: 35.96% 
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Percentage of jobs coining from other sites: 87.41% 
Percentage of besteffort jobs: 50.01% 

Percentage of time spent in besteffort jobs compared to the work time: 23.38% 



• Users: 



Number of users: 213 

Percentage of users coming from other sites: 82.63% 

Distribution in time of thie different resource's states per weel< for Rennes 






Jobs 




Idle 




Work 




Absent 




Suspected 




Dead 



Weeks 



Figure 22: Global diagram with dead time for Grid'5000's Rennes site 
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Figure 23: Global diagram without dead time for Grid'5000's Rennes site 
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Figure 24: Median resource diagram for Grid'5000's Rennes site 
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Maximal resource's states per week for Rennes - number 371 
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Figure 25: Maximal resource diagram for Grid'5000's Rennes site 



JL From Figures 24 and 25, we see a really heterogeneous usage of the resources (proba- 
bly) due to the heterogeneity of the nodes. This site is the biggest one in terms of resources 
and its activity percentage is really on the average. Due to its size, it allows bigger reserva- 
tion. So, it has the higher mean number of resources per reservation (27.32) but one of the 
smallest number of reservations for the whole year. 



3.7 Usage of Grid'5000's Sophia site in 2008 

• Platform and resources: 

— Maximal number of resources (cores): 568 

— Mean time spent in each state for all the resources, in percentage: 

* Dead: 2.04% 

* Suspected: 0.37% 

* Absent: 1.98% 

* Work: 78.25% 

— Real percentage of work time (without taking into account the time when the 
resources are dead or absent): 81.51% 

• Jobs: 

— Number of jobs (reservations): 58142 

— Mean time of a job: 8767.35 s. (2 hours 26 minutes and 7 seconds) 

— Maximal duration: 485924 s. (5 days 14 hours 58 minutes and 44 seconds) for 
job number 345959 

— Mean number of resources (cores) per job: 22.14 
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— Percentage of deploy jobs: 8.76% 

— Percentage of time spent in deploy jobs compared to the work time: 25.64% 

— Percentage of jobs coming from other sites: 87.24% 

— Percentage of besteffort jobs: 54.36% 

— Percentage of time spent in besteffort jobs compared to the work time: 19.57% 
• Users: 

— Number of users: 187 

— Percentage of users coming from other sites: 85.03% 



Distribution in time of ttie different resource's states per weei( for Soptiia 
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Figure 26: Global diagram with dead time for Grid'5000's Sophia site 
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Distribution of unoccupied time and activity per weei< lor Sopiiia 
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Figure 27: Global diagram without dead time for Grid'5000's Sophia site 



Median resource's states per weel< for Sophia - number 148 
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Figure 28: Median resource diagram for Grid'5000's Sophia site 
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Maximal resource's states per week for Sophia - number 542 
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Figure 29: Maximal resource diagram for Grid'5000's Sophia site 

This site has the smallest number of jobs for the whole year compared to the other 
Grid'5000 sites. But, due to a high mean number of resources per reservation, it gets the 
highest percentage of activity (81.51%). This is illustrated on Figures 26 and 27. 




3.8 Usage of Grid'5000's Toulouse site in 2008 

• Platform and resources: 

— Maximal number of resources (cores): 434 

— Mean time spent in each state for all the resources, in percentage: 

* Dead: 5.00% 

* Suspected: 1.22% 

* Absent: 14.86% 

* Work: 49.42% 

— Real percentage of work time (without taking into account the time when the 
resources are dead or absent): 61.67% 

• Jobs: 

— Number of jobs (reservations): 166191 

— Mean time of a job: 2211.80 s. (36 minutes and 52 seconds) 

— Maximal duration: 1080018 s. (12 days 12 hours and 18 seconds) for job number 
94101 

— Mean number of resources (cores) per job: 6.29 

— Percentage of deploy jobs: 1.47% 



RR n° 7481 



28 



Orgerie & Lefevre 



Percentage of time spent in deploy jobs compared to the work time: 28.37% 
Percentage of jobs coming from other sites: 96.15% 
Percentage of bestefFort jobs: 6.86% 

Percentage of time spent in besteffort jobs compared to the work time: 24.85% 



• Users: 



Number of users: 160 

Percentage of users coming from other sites: 81. 



Distribution in time of thie different resource's states per weel< for Toulouse 
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Figure 30: Global diagram with dead time for Grid'5000's Toulouse site 
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Distribution of unoccupied time and activity per weeic for Toulouse 
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Figure 31: Global diagram without dead time for Grid'5000's Toulouse site 



Median resource's states per week for Toulouse - number 223 
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Figure 32: Median resource diagram for Grid'5000's Toulouse site 
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Maximal resource's states per week for Toulouse - number 423 
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Figure 33: Maximal resource diagram for Grid'5000's Toulouse site 

Toulouse has the shortest reservations with an average around 36 minutes. During 
the last two weeks of the year, the site is used at its full capacity (about 98% of activity). 
This site is one of the smallest Grid'5000 sites. 




3.9 Results summary 

Tables 1, 2, 3, 4, summarize the main observed statistics in terms of resources, jobs and users. 
These results show that the GridSOQO platform faces some heterogeneous usage depending 
on the involved sites. 



• Platform and resources: 



Site 


Number of resources (cores) 


Percentage of 'real' activity 


Bordeaux 


650 


53.20% 


Lille 


618 


72.89% 


Lyon 


322 


69.27% 


Nancy 


574 


60.08% 


Orsay 


684 


57.82% 


Rennes 


714 


64.58% 


Sophia 


568 


81.51% 


Toulouse 


434 


61.67% 


Total Grid'5000 


4564 


64.67% 



Table 1: Resource-related statistics 



• Jobs: 
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Number of 


Mean number 


Mean duration 


Jobs coming 


Site 


jobs 


of resources 


of a job 


from other 




(reservations) 


per job 


(seconds) 


sites 


Bordeaux 


356222 


7.44 


2473.38 


83.28% 


Lille 


344538 


8.11 


3154.58 


38.97% 


Lyon 


138217 


4.39 


3723.55 


87.07% 


Nancy 


74592 


14.63 


8912.82 


20.11% 


Orsay 


92862 


14.58 


6246.07 


96.28% 


Rennes 


58843 


27.32 


7069.33 


87.41% 


Sophia 


58142 


22.14 


8767.35 


87.24% 


Toulouse 


166191 


6.29 


2211.80 


96.15% 



Table 2: Job-related statistics 





Percentage 


Time spent 


Percentage 


Time spent 


Percentage 


Time spent 


Site 


of deploy 


in deploy 


of besteffort 


in besteffort 


of default 


in default 




jobs 


jobs 


jobs 


jobs 


jobs 


jobs 


Bordeaux 


0.97% 


22.33% 


58.03% 


34.90% 


41.00% 


42.77% 


Lille 


0.61% 


7.24% 


19.10% 


37.88% 


80.29% 


54.88% 


Lyon 


3.88% 


49.69% 


44.00% 


24.14% 


52.12% 


26.17% 


Nancy 


4.23% 


29.95% 


76.16% 


17.05% 


19.61% 


53.00% 


Orsay 


3.59% 


38.98% 


38.91% 


27.17% 


57.50% 


33.85% 


Rennes 


7.84% 


35.96% 


50.01% 


23.38% 


42.15% 


40.66% 


Sophia 


8.76% 


25.64% 


54.36% 


19.57% 


36.88% 


54.79% 


Toulouse 


1.47% 


28.37% 


6.86% 


24.85% 


91.67% 


46.78% 



Table 3: Job-related statistics 



• Users: 



Site 


Number of users 


Users from other sites 


Bordeaux 


223 


87.00% 


Lille 


198 


65.66% 


Lyon 


169 


73.37% 


Nancy 


157 


83.44% 


Orsay 


172 


76.16% 


Rennes 


213 


82.63% 


Sophia 


187 


85.03% 


Toulouse 


160 


81.88% 



Table 4: User-related statistics 
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4 Conclusions 

We have observed an activity increase for all the Grid'5000 sites compared to 2007 year [OL09]. 
The global activity of the platform has grown from 40% to 65%. 2007 was still a develop- 
ment year for Grid'5000, whereas in 2008, the platform is fully working and it reaches its 
cruising speed. 
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Distribution in time of the minimal resource's states per week for Bordeaux - number 58 
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Minimal resource's states per weel< for Lille - number 49 
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Minimal resource's states per week for Lyon - number 50 
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Minimal resource's states per weel< for Nancy - number 252 
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Minimal resource's states per \Neek for Orsay - number 607 
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Minimal resource's states per week for Rennes - number 211 
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Minimal resource's states per week for Sophia - number 92 
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Minimal resource's states per week for Toulouse - number 101 
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